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ABSTRACT 


The  objective  of  this  work  was  to  determine  whether  a  direct  statistical  or 
stochastic  relationship  between  the  following  systemic  characteristics  of  dedicated  built- 
in-test-equipment  (BITE)  could  be  derived  and  quantified:  annual  maintenance  costs, 
user  acceptance,  operational  availability,  and  “complexity”  (defined  as  total  number  of 
sensor  interfaces  per  system).  Three  systems  of  ascending  degrees  of  complexity  from  the 
USAF  F-15A/BC/D,  O/A-lOA,  and  C-5A/B/C  were  analyzed,  and  based  upon  raw  data 
acquired  from  field  operating  units  and  fleet-wide  maintenance  data  collection  a  model 
was  constructed  to  derive  constraints  on  a  postulated  “best-fit”  interdependence  between 
these  four  characteristics.  The  chief  finding  was  that  BITE  reliability  and  minimal 
intrinsic  system  maintenance  burden  were  the  prime  detenninants  of  user  acceptance  and 
therefore  system  success.  A  corollary  finding  was  that  the  number  of  data  interfaces  (or 
sensors)  was  mathematically  irrelevant  to  user  acceptance,  suggesting  that  condition- 
based  monitoring  schemas  are  feasible  provided  that  BITE  system-level  reliability  is 
maximized  with  a  minimal  maintenance  burden  placed  on  the  user  community.  Sensor 
redundancy  to  achieve  this  goal  was  the  suggested  method.  This  model  may  be  used  as  an 
objective  criterion  for  evaluating  future  BITE  system  procurement  proposals,  a  critical 
concern  for  the  emerging  predictive/condition-based  maintenance  paradigms  currently 
favored  by  the  Department  of  Defense,  NASA,  and  other  Federal  and  commercial 
agencies. 
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EXECUTIVE  SUMMARY 


This  paper  examines  the  utility  and  user  acceptance  of  three  dedicated  aircraft 
built-in-test  equipment  (BITE)  suites  with  respect  to  documented  maintenance  man-hour 
expenditures  over  a  twenty-year  period  as  well  as  a  survey  of  system  users.  The  objective 
was  to  detennine  the  factor(s)  that  influence  BITE  acceptance  by  end-users  (in  this  case, 
aircraft  maintainers)  and  thereby  provide  design  heuristics  and  other  relevant  guidance 
for  future  system  design.  BITE  effectiveness  is  a  central  tenet  of  the  emerging  DoD-wide 
condition-based  maintenance  (CBM)  paradigm,  and  therefore  maximizing  the  quality  and 
utility  of  data  return  from  such  systems  is  critical  for  the  initiative's  success. 

The  chief  finding  was  that  user  acceptance  of  BITE  equipment  is  inversely  related 
to  the  amount  of  effort  required  by  users  to  maintain  the  equipment  itself.  BITE  is 
unusual  in  that  its  primary  end  users  are  also  usually  the  same  people  who  maintain  it, 
and  perceived  value  of  its  functionality  is  critically  dependent  upon  its  perceived,  more  so 
than  empirically  detennined,  reliability.  One  surprising  ancillary  finding  was  that  this 
user  trust  seemed  to  be  statistically  independent  of  the  number  of  BITE-monitored 
subsystems,  which  is  encouraging  for  the  future  of  CBM  but  also  may  require  greater 
investment  in  BITE  system  physical  infrastructure  and  reliability  engineering  during 
system  development. 

Recommended  design  heuristics  include  providing  redundant  sensors  for  critical 
weapon  system  functions  and  explicitly  requiring  that  future  BITE  systems  must  not 
require  more  maintenance  than  the  system(s)  that  they  monitor.  Future  research  may 
reveal  other  useful  aspects  of  this  issue  that  will  improve  operational  availability  while 
minimizing  sustainment  costs  for  all  highly  integrated  manned  and  unmanned  DoD 
weapons  systems. 
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I.  INTRODUCTION  AND  PROBLEM  DEFINITION 


A.  BITE  BACKGROUND  INFORMATION 

Built-in-test-equipment  (BITE)  as  a  distinct  sub-system  design  discipline  began 
its  most  rapid  evolutionary  phase  primarily  in  the  commercial  aviation  sector  during  the 
1980s,  although  its  roots  can  be  found  in  the  early  manual  “push  to  test”  functions  used 
by  many  aircraft  systems  designed  in  the  1950s.  Early  features  were  designed  to  provide 
pre-flight  or,  occasionally,  in-flight  equipment  confidence  checks  for  aircrews  in  order  to 
more  readily  and  rapidly  discriminate  between  malfunctions  of  indicating  equipment  and 
mechanical  systems,  or  to  identify  true  abnormal  performance.  Use  of  these  features  by 
maintenance  personnel  was  an  incidental  byproduct  that  later  became  dominant  in 
aviation  as  a  means  to  minimize  procuring  and  maintaining  ancillary  ground  test 
equipment.  In  the  author's  opinion,  BITE  is  also  perceived  by  many  planners  as  a  means 
to  minimize  personnel  training  requirements  by  incorporating  as  much  self-diagnostic 
capability  as  possible  into  weapons  system  design.  BITE  is  also  now  widely  employed  in 
ground,  maritime,  and  space-borne  weapons  systems  (though  restricted  to  ground  systems 
for  the  latter  in  its  traditional  sense)  for  both  operational  confidence  checks  and  as  a 
central  tool  for  malfunction  isolation  and  rapid  restoration  of  full  functionality. 

The  Aeronautical  Radio  Corporation  of  America,  usually  referred  to  as  ARINC, 
published  the  first  standard  for  airborne  BITE,  ARINC  604  in  1985  (Spitzer,  2007).  In 
addition  to  defining  specific  serial  data  protocols,  this  standard  for  the  first  time  defined  a 
centralized  fault  display  system  that  would  permit  aircrews  and  maintainers  to  view 
failure  messages  from  a  wide  variety  of  equipment  at  one  location.  Prior  to  this  first 
attempt  at  standardization,  the  US  Air  Force  had  already  developed  and  fielded  several 
dedicated  BITE  systems  with  mixed  results,  which  will  be  further  explored  later. 

Modern  built-in-test  (BIT)  systems  can  be  broadly  placed  under  two  primary 
categories  based  on  their  operational  modes  and  methodologies:  active  and  passive. 

Active  systems  initiate  test  routines  in  subsystems  or  components  and  report 
results  to  operators  or  maintainers.  Active  BIT  is  usually  confined  to  a  specific  platform 
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system  in  order  to  prevent  spurious  inputs  from  otherwise  non-function-essential 
equipment  from  interfering  with  (or  disabling)  the  system  during  normal  operations.  A 
prime  example  of  this  design  philosophy  is  the  extensive  built-in-test  routine  employed 
by  most  aircraft  autopilots.  However,  platforms  of  more  recent  vintage  such  as  the  C-17 
do  employ  multi-system  distributed  BITE  as  a  consequence  of  the  highly  integrated 
nature  of  the  weapons  system.  The  C-17  flight  control  system  BIT  requires  several 
minutes  for  full  execution  and  tests  every  critical  subsystem  related  to  aircraft  control  and 
engine  performance  on  the  aircraft. 

Passive  systems  monitor  and  record  the  performance  of  various  platfonn 
subsystems,  and  the  results  are  examined  after  mission  execution  for  evidence  of 
monitored  system  malfunctions  as  well  as  archived  for  historical  trend  data  analysis. 
Most  passive  BITE  does  not  provide  a  real-time  display  to  operators  during  mission 
execution,  and  consequently  its  target  user  community  is  restricted  to  maintainers.  An 
exception  to  this  design  philosophy  is  the  C-5  MADAR  system. 

The  pioneering  C-5A  MADAR  (Malfunction  Analysis,  Detection,  and  Recording) 
system  described  by  Mash  (1968)  was  an  early  attempt  to  implement  comprehensive 
weapons  system  platform-level  condition  monitoring.  As  originally  designed,  the  system 
consisted  of  40  modules,  or  line  replaceable  units  (LRUs)  and  periodically  sampled 
nearly  800  test  points  during  flight.  Certain  critical  aircraft  performance  parameters  such 
as  engine  vibration  were  displayed  during  flight  on  an  oscilloscope  and  monitored  by  the 
flight  engineer.  Given  its  large  number  of  components,  MADAR  was  (and  still  is) 
arguably  the  most  complex  discrete  subsystem  on  the  C-5A,  and  perhaps  unsurprisingly 
also  incurred  a  large  number  of  maintenance  actions  not  only  for  the  systems  it 
monitored,  but  for  its  own  upkeep.  Its  overall  utility  and  added  value  to  the  C-5A  mission 
was  therefore  open  to  question  on  many  levels.  Later  upgrades  to  the  system  were 
apparently  focused  on  adding  additional  monitoring  capability  (currently  2000  test 
points)  and  improving  the  both  the  reliability  of  the  LRUs  which  provide  the  human- 
machine  interface  and  presentation  methods  more  so  than  improving  user-level 
confidence  in  system  accuracy. 
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Figure  1.  Avionics  technician  prepares  to  troubleshoot  a  C-5A.  (From:  G.  Wantland, 
2007) 


This  thesis  examines  dedicated  BITE  of  different  types,  complexity,  and 
objectives  across  three  USAF  weapons  systems,  and  derives  a  set  of  boundary  conditions 
and/or  figurc(s)  of  merit  that  may  yield  the  best  value  for  warfighters  during  future  BITE 
design  efforts  while  simultaneously  conserving  system  acquisition  and  sustainment  costs. 

B.  RESEARCH  QUESTIONS 

At  what  point,  and  based  on  which  factors,  does  maintenance  of  built-in-test  and 
monitoring  (BITE)  subsystems  obviate  or  negatively  impact  weapons  system  life-cycle 
cost  and  operational  availability?  If  this  question  has  a  quantifiable  answer,  then  the 
following  design  heuristic  emerges:  “BITE  must  not  be  more  time-consuming  or 
expensive  to  maintain  than  the  system(s)  it  monitors.”  An  analogous  statement  is 
frequently  used  as  contractual  language  for  the  acquisition  of  new  space  systems  that 
replace  legacy  equipment,  which  can  be  paraphrased  as  follows:  new  systems  must  not  be 
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more  difficult  to  maintain  than  the  old.  Specifying  BITE  in  this  context  is  a  narrowing  of 
this  paradigm;  however,  this  improved  focus  also  offers  the  opportunity  to  introduce  far 
more  rigorous  metrics  as  well  as  design  guidelines,  as  will  be  seen  later. 

Is  there  an  empirical  relationship  between  the  overall  systemic  complexity  of 
BITE  subsystems,  the  analogous  complexity  of  monitored  subsystems,  and  total  life-cycle 
cost  for  a  given  integrated  weapons  system  so  equipped?  This  question  is  designed  to 
determine  whether  a  critical  point,  or  intersection  of  the  stated  variables,  exists  beyond 
which  BITE  does  not  add  value  to  weapons  systems  maintenance  but  is  either  neutral  or  a 
burden  to  sustainment.  ('Value’  in  this  context  is  defined  as  the  system's  capability  to 
enable  more  rapid  and  accurate  repair  of  monitored  systems  versus  its  intrinsic 
maintenance  burden.)  Conversely,  an  insufficiently  sophisticated  or  capable  BITE  suite 
may  add  little  if  any  value  whatsoever,  and  again  act  as  a  negative  factor  for  sustainment. 
Both  regions  of  this  potential  operational  space  will  be  examined. 

Can  a  mathematical  model  representing  interfaces  and  monitoring  capabilities  be 
constructed  to  optimize  BITE  infrastructure  during  the  design  phase  to  provide  maximum 
operational  availability  while  minimizing  life-cycle  costs?  The  desired  end  result  of  this 
study  is  a  model  capable  of  defining  optimal  interface  levels,  methods  and  operational 
reliability  values  framed  by  measure(s)  of  effectiveness  for  user  acceptance  that  can 
define  BITE  capabilities  within  these  constraints  as  a  function  of  total  weapons  system 
lifecycle  support  expenditures.  Subjective  observations  by  the  author  during  more  than  20 
years  of  direct  experience  in  aircraft  maintenance  led  to  the  hypothesis  that  an  optimal 
balance  of  BITE  capabilities  as  described  may  be  achievable,  but  that  no  such  analysis 
has  been  performed  to  date  that  may  facilitate  development  of  systems  that  meet  these 
criteria. 


C.  RESEARCH  METHODOLOGY 

After  consideration  of  possible  relevant  factors,  five  potential  decision  variables 
were  selected,  with  the  presumption  that  interdependencies  would  be  identified  between 
at  least  some  of  these  factors  and  provide  insight  as  part  of  the  research.  Three  Air  Force 
aircraft  types  were  selected  as  the  platforms  to  be  examined  since  extensive,  long-term 
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detailed  statistical  maintenance  and  reliability  data  was  readily  available  through 
restricted  Internet  access.  Additionally,  the  author  has  had  personal  experience  with  all 
three  systems  and  this  fact  facilitated  acquisition  and  interpretation  of  all  data.  The  ‘end 
users’  cited  are  USAF  avionics  maintenance  personnel. 

Man-hour  ratio.  This  was  defined  as  the  percentage  of  man-hours  expended  to 
maintain  a  given  BITE  system  divided  by  the  total  number  of  man-hours  needed  to 
maintain  the  entire  weapons  system  avionics  suite,  or  in  other  words  the  fraction  of  direct 
labor  costs  directly  applied  to  maintain  a  system  used  to  improve  maintenance  of  other 
aircraft  systems.  Avionics  systems  identified  for  determining  total  workload  belonged  to 
the  following  two-digit  work  unit  code  groups:  46,  51,  52,  55  (the  group  that 
encompasses  dedicated  BITE  systems),  71,  72,  and  74.  The  metric  was  obtained  by 
dividing  the  sum  of  man-hours  expended  on  maintenance  of  these  systems  over  a  20-year 
time  span  by  the  man-hours  expended  on  the  code  55  group.  These  systems  are 
maintained  by  enlisted  personnel  in  a  family  of  career  fields  generically  referred  to  as 
aircraft  guidance  and  control.  Data  was  acquired  from  the  USAF  Reliability  Engineering 
Management  Infonnation  System  (REMIS)  with  the  assistance  of  Warner-Robins  AFB 
personnel,  and  is  provided  in  appendices  1,  2  and  3. 

Number  of  data  interfaces.  Defined  as  the  total  number  of  sensor/BITE 
connections  as  well  as  BITE  LRU-to-LRU  connections  for  a  given  system.  Power  and 
other  utility  interfaces  were  not  considered  since  a  supporting  study  objective  was  to 
qualitatively  assess  data  return  to  the  user. 

Perceived  utility  by  end  user.  Although  subjective  in  nature,  this  variable  is  a  vital 
consideration  in  BITE  system  design.  'Perceived  utility'  in  this  context  is  defined  as  how 
useful  the  users  believe  the  subject  system  to  be  as  a  troubleshooting  tool.  Data  was 
acquired  for  each  subject  system  via  question  #2  of  a  standardized  survey  (see  Appendix 
4)  that  was  distributed  to  members  of  the  A-10,  F-15,  and  C-5  maintenance  communities. 
Please  refer  to  the  end  of  this  section  for  further  discussion  of  the  survey  methods 
employed. 
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Perceived  reliability  by  end  user.  Again  subjective  yet  critical  for  application 
success,  this  data  was  also  acquired  via  the  aforementioned  survey  and  represents  the 
average  of  responses  to  question  #3.  ’Perceived  reliability'  is  defined  as  the  users'  core 
impression  of  how  frequently  the  subject  system  itself  malfunctions,  which  clearly  will 
also  affect  the  users'  level  of  trust  in  system  outputs. 

Perceived  maintenance  burden  by  end  user.  This  final  subjective  variable 
measures  level  of  effort  needed  to  resolve  BITE  malfunctions  relative  to  the  amount  of 
time  users  spend  resolving  other  aircraft  system  malfunctions,  which  is  another  critical 
component  of  operational  suitability.  Source  data  was  question  #4  of  the  survey. 

Questions  1  and  5  of  the  survey  were  ultimately  not  included  in  the  data  analysis. 
Question  1  measured  the  perceived  amount  of  time  spent  by  technicians  maintaining  the 
subject  system,  but  given  the  availability  of  objective  data  from  REMIS  this  information 
was  redundant  and  less  reliable.  Likewise,  question  #5  was  originally  included  to  provide 
a  figure  of  merit  for  the  level  of  experience  of  respondents,  but  the  data  proved 
unnecessary  for  the  analytical  effort  since  a  core  assumption  of  BITE  is  that  user  level  of 
experience  is  minimal. 

Variables  1  and  2  are  physical  system  attributes  that  can  be  considered  as  figures 
of  merit  that  represent  gross  systemic  complexity.  Variables  3,  4  and  5  represent  the 
perceived  amount  of  trust  that  the  end  users  have  in  the  system’s  ability  to  perfonn  its 
desired  function  as  well  as  the  level  of  effort  needed  to  maintain  its  operation.  Since 
BITE  is  not  only  used  but  also  maintained  by  its  end  users,  this  variable  set  measures  its 
user  acceptance  in  terms  of  its  effects  on  the  user’s  overall  workload,  a  unique,  almost 
recursive  characteristic  of  this  equipment  class.  Question  5  was  used  as  a  root  measure  of 
the  respondent’s  breadth  of  experience. 

Much  of  this  study  is  centered  on  the  core  issue  of  user  trust  of  a  given  system. 
According  to  Langford  (2007),  the  concept  of  'trust'  with  respect  to  weapons  system 
performance  can  be  defined  thusly:"  ...trust  can  be  considered  as  follows:  Trust  factors 
into  four  uniquely  definable  and  quantifiable  variables.  They  are  knowledge  (the  norms  of 
operation,  the  expected  actions,  the  rights  and  pennissions  granted);  truth  (the 
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verifiability  of  data  supplied);  belief  (consistency  of  patterns  and  standards);  and 
retribution  (one’s  stake  or  ante).  Trust  is  the  additive  aggregation  of  knowledge,  truth, 
belief,  and  retribution.  Trustworthiness  is  the  degree  to  which  trust  is  established."  The 
data  acquired  was  selected  in  accordance  with  this  definition,  and  the  results  of  the  study 
are  framed  in  this  context. 

All  survey  data  was  normalized  by  assigning  numerical  values  to  each  of  the  five 
possible  responses,  summing  the  total  for  each  question,  and  subsequently  dividing  this 
number  by  the  number  of  samples  (N),  which  were  9,  1 1  and  19  for  the  A- 10,  C-5  and  F- 
15  responses  respectively. 

The  C-5  surveys  were  distributed  and  collected  by  the  author  in  person  at  Travis 
AFB,  CA  in  November  2006.  A- 10  surveys  were  distributed  electronically  at  Pope  AFB, 
NC  and  Davis-Monthan  AFB,  AZ  to  flightline  aircraft  maintainers  by  locally  assigned 
members  of  the  Air  Force  Engineering  and  Technical  Service  (AFETS),  an  independent 
USAF  agency,  and  returned  directly  to  the  author.  F-15  surveys  were  similarly  distributed 
and  collected  by  an  AFETS  representative  at  Elmendorf  AFB,  AK. 

All  data  collected  was  processed  using  Microsoft  Excel®  to  perform  all  required 
calculations  and  generate  graphical  data  presentations  for  subsequent  analysis  of  results. 

D.  STUDIED  BITE  SYSTEM  DESCRIPTIONS 

Selected  systems  were  chosen  based  on  their  relative  complexity  and  scope  of 
service.  Based  on  the  author's  experience  with  approximately  20  types  of  USAF  aircraft, 
they  represent  a  rough  continuum  of  interface  levels  and  capabilities  that  accurately 
reflect  the  current  state  of  airborne  BITE  functionality.  Additionally,  the  F-15  ASP 
design  philosophy  is  extensively  emulated  on  other  aircraft,  and  the  A- 10  TEMS  system 
is  also  used  on  the  KC-135R  and  F-l  17A  aircraft.  The  least  complex  system  in  terms  of 
operations,  LRU  count,  and  interfacing  is  the  F-15  Avionics  Status  Panel  (ASP).  The  A- 
10  Turbine  Engine  Monitoring  System  (TEMS)  is  an  intennediate  example  for  purposes 
of  this  analysis,  and  the  C-5  MADAR II  was  the  most  complex  system  examined. 
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The  F-15  ASP  provides  “go/no-go”  indications  to  ground  maintenance  personnel 
for  specific  aircraft  systems  via  a  simple  set  of  black/white  magnetically  latched  ball 
indicators  mounted  on  the  panel  itself,  which  is  located  in  the  aircraft’s  nose  wheel  well 
and  therefore  not  accessible  to  the  aircrew  during  flight.  Dedicated  system  LRUs  are 
limited  to  the  status  panel  itself  as  described  and  a  BIT  reset  panel  in  the  cockpit.  The 
ASP  receives  a  serial  digital  data  stream  from  the  aircraft  central  computer  (CC),  and 
upon  receipt  of  a  failure  status  word  for  a  given  system  or  aircraft  physical  region  will  set 
the  appropriate  indicator.  The  ASP  has  72  such  indicators,  but  usage  is  controlled  by  CC 
operational  flight  program  configuration;  many  are  not  used  or  are  disabled  as  nuisance 
indications  (Black,  2007). 

The  A- 10  TEMS  continuously  acquires  and  records  performance  monitoring  data 
for  the  aircraft’s  two  TF-34-100A  engines  as  well  as  selected  environmental  parameters 
during  flight.  The  system  is  also  used  with  an  auxiliary  real-time  ground  terminal  to 
perform  engine  maintenance  actions.  No  indications  are  provided  to  the  pilot;  indeed,  no 
real-time  data  display  exists  at  all  unless  the  auxiliary  terminal  is  employed  during 
ground  maintenance  actions.  However,  stored  data  is  downloaded  from  each  aircraft  once 
per  day  and  input  into  the  USAF  Consolidated  Engine  Management  System  (CEMS)  for 
trend  analysis.  CEMS  also  provides  notification  to  maintenance  personnel  of  potential 
system  or  engine  discrepancies  for  specific  aircraft  during  this  process.  TEMS  consists  of 
21  dedicated  LRUs  and  accepts  data  from  10  legacy  engine  indication  systems. 

C-5  MADAR  II  is  a  comprehensive  platform  status  monitoring  system  that 
continuously  acquires  data  on  nearly  every  C-5  subsystem  and  provides  a  real-time 
interactive  display  to  the  aircrew  during  flight.  MADAR  consists  of  40  dedicated  LRUs 
that  monitor  approximately  2000  discrete  test  points  throughout  the  aircraft.  The  majority 
of  these  LRUs  are  interface/signal  conditioning  devices  that  provide  direct  connections  to 
the  monitored  systems  and  convert  analog  information  into  a  serial  digital  data  stream  for 
subsequent  processing  by  the  recording/display  LRU  group.  Like  the  A- 10  TEMS,  data  is 
continuously  recorded,  retrieved  and  archived  in  a  ground  database  which  also  provides 
malfunction  notifications  to  maintenance  personnel.  One  major  function  of  the  system  is 
to  monitor  TF-39  engine  vibration  in  real  time  during  flight.  Additionally,  MADAR 
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provides  most  data  inputs  to  the  aircraft  flight  data  recorder.  Therefore,  unlike  other 
BITE  systems  described  thus  far,  MADAR  is  essential  for  safe  operation  of  the  aircraft 
since  these  particular  features  are  considered  flight-critical  functions. 
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II.  ANALYSIS  AND  INITIAL  FINDINGS 


A.  RESEARCH  APPROACH 

The  gross  function  of  the  model  is  to  compare  empirical  values  for  the  physical 
attributes  of  these  systems  ("objective  variables")  to  measures  of  user  acceptance 
("subjective  variables").  The  objective  of  the  model  is  to  determine  relationships,  if  any, 
between  the  physical  design  attribute  of  interface  quantity,  documented  reliability,  and 
measures  of  user  acceptance.  These  variables  are  defined  as  follows: 

Objective  Variables  (Units) 

M  =man-hour  ratio  (BITE  system  maintenance  hours/total  avionics  system 
maintenance  hours) 

I=total  number  of  data  interfaces  (positive  integer) 

Subjective  Variables  (Units) 

U=perceived  utility  (real  number  between  1  and  5,  large  values  good,  source  data 
was  question  #2  of  survey) 

R=perceived  reliability  (real  number  between  1  and  5,  large  values  good,  source 
data  was  question  #3  of  survey) 

B=perceived  maintenance  burden  (real  number  between  0  and  1,  large  values 
good,  source  data  was  the  average  value  of  question  #4  of  the  survey  divided  by  five  to 
yield  a  zero  to  one  scaling  factor) 

For  each  system,  a  logarithmic  plot  of  M  versus  I  was  derived.  Likewise,  R  versus 
U  was  plotted,  with  the  additional  condition  that  B  was  used  as  a  coefficient  for  U;  actual 
values  examined  were  R  versus  B(U).  The  heuristic  applied  to  this  action  is  that 
perceived  utility  of  a  given  system  for  its  design  function  is  inversely  proportional  to  the 
amount  of  effort  required  to  maintain  it.  Further  investigation  also  generated  plots  of  M 
versus  R  and  M  versus  B  for  comparison. 
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To  establish  a  baseline  for  analysis,  an  "ideal"  system  was  constructed  for 
comparison  with  the  following  measured  values: 


Table  1.  BITE  system  figures  of  merit. 


IDEAL 

Raw 

Log 

M  = 

0.01 

-2 

1  = 

5000 

3.69897 

R  = 

5 

0.69897 

B  = 

1 

0 

U  = 

5 

0.69897 

B(U)  = 

5 

0.69897 

A-10  TEMS 

Raw 

Log 

M= 

0.063495358 

-1.19726 

1  = 

31 

1.462398 

R  = 

3.333333333 

0.522879 

B  = 

0.666666667 

-0.17609 

U  = 

3 

0.477121 

B(U)  = 

2 

0.30103 

C-5  MADAR 

Raw 

Log 

M= 

0.190651494 

-0.71976 

1  = 

2018 

3.304921 

R  = 

2.363636364 

0.373581 

B  = 

0.509090909 

-0.2932 

U  = 

2.272727273 

0.356547 

B(U)  = 

1.157024793 

0.063343 

F-15  ASP 

Raw 

Log 

M= 

0.042565579 

-1.37094 

1  = 

74 

1.869232 

R  = 

3.210526316 

0.506576 

B  = 

0.431578947 

-0.36494 

U  = 

3 

0.477121 

B(U)  = 

1.294736842 

0.112182 

M,  the  fraction  of  man-hours  expended  for  system  maintenance  was  set  at  1%  for 
the  ideal  case,  almost  certainly  the  best  possible  achievable  value  under  real  conditions. 
DoD  weapons  system  operational  requirement  documents  (ORDs)  and  the  more  recent 
capability  development  documents  (CDD)  frequently  cite  objective  operational 
availability  (A0)  targets  of  99%  for  mission-critical  systems.  Complex  'systems  of 


12 


systems'  such  as  aircraft  and  space  vehicles  require  the  most  accurate  and  dependable  in 
situ  BITE  capability  possible  to  achieve  such  goals,  which  in  turn  implies  that  BITE 
system  maintenance  expenditures  should  be  absolutely  minimal  due  to  its  presumed 
reliability.  The  number  of  interfaces  (I)  was  arbitrarily  and  initially  set  at  5000,  more  than 
twice  the  C-5  MADAR  value  but  reasonable  for  advanced  next-generation  equivalent 
systems.  It  was  later  found  that  the  value  of  I  was  not  significant  for  the  model.  Perceived 
reliability  (R),  maintenance  burden  (B),  and  system  utility  (U)  were  all  set  at  the  greatest 
possible  favorable  values. 

B.  DATA  ANALYSIS 

Graphical  methods  were  selected  as  the  preferred  tool  to  examine  relationships 
between  these  data  sets.  It  must  be  emphasized  that,  as  original  research,  this  data  has  not 
been  compared  against  any  findings  derived  from  similar  studies  since  no  equivalent 
research  was  found,  and  therefore  statistical  error  limits  are  not  known.  Initial 
comparison  of  the  "ideal"  system  to  the  A- 10,  C-5  and  F-15  studied  BITE  systems 
yielded  the  following  curves  (10  x  log  y-axis  used  to  separate  values): 
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All  Systems  M,  I,  R,  B(U) 


A-10-*— C-5  F-15  Ideal 


Figure  2.  BITE  Performance  parameters  compared  to  ideal  values. 

This  comparison  did  not  appear  to  be  particularly  useful,  although  it  did  illustrate 
two  later  critical  points: 

M  and  I  (the  two  left-most  data  points)  should  be  considered,  at  least  during 
preliminary  analysis,  separately  from  R  and  B(U). 

The  R-B(U)  line  segment  of  the  ideal  system  satisfied  the  equation  y=x,  while  the 
real  systems  did  not,  indicating  that  the  relationship  between  reliability  and  perceived 
user  utility  was  not  'simple'. 

Since  the  number  of  interfaces  varied  considerably  among  the  systems,  a  second 
plot  was  generated  omitting  I  and  comparing  the  remaining  three  parameters,  as  shown  in 
Figure  3. 
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This  plot  suggested  the  possibility  that  a  significant  correlation  may  exist  between 
M,  the  fraction  of  man-hours  used  to  maintain  BITE  and  the  user  acceptance  measures  R 
and  B(U).  Further  examination  of  the  ideal  system’s  behavior  was  warranted. 

The  relationship  between  M  and  the  other  variables  was  obviously  of  interest.  M 
is  a  direct  indicator  of  a  BITE  system's  maintenance  requirements,  and  understanding  its 
interaction  with  other  system  attributes  is  important  with  respect  to  research  question  #1. 
Figures  3  through  5  on  the  following  page  depict  these  relationships: 
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Figure  4.  M  vs.  I  for  ideal  system  (1=5000). 
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Figure  6.  M  vs.  B(U)  for  ideal  system. 

This  examination  revealed  that  for  M=0.01,  the  relationship  between  the  decimal 
logarithm  of  M  and  those  of  I,  R,  and  B(U)  can  be  generically  expressed  as 

y=kx-(k+2)  (eq.  1) 

Interestingly,  this  relationship  remained  valid  for  all  values  of  I,  suggesting  that 
the  number  of  data  interfaces  has  no  inherent  effect  on  system  maintenance  time.  At  first 
glance  that  statement  seems  oxymoronic  at  best,  but  later  discussion  of  design 
implications  will  illustrate  the  importance  of  this  finding,  which  bodes  well  for  future 
BITE  architectures. 

As  previously  described,  the  second  key  relationship,  R  vs.  B(U),  is  trivial  in  the 
ideal  case  (y=Jt),  but  more  complex  in  real  systems.  Consider  the  following  tabular 
comparisons  of  R  and  B(U): 


Table  2.  R  and  B(U)  values  for  subject  systems. 


Platform/System 

Log(R) 

Log(B(U)) 

A-10/TEMS 

3.333333333 

2 

C-5/MADAR 

2.363636364 

1.157024793 

F-15/ASP 

3.210526316 

1.294736842 
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Peak  values  of  R  obviously  do  not  correlate  with  identical  values  of  B(U).  The 
manifest  reason  for  this  discontinuity  is  that  B(U)  represents  user  effort  required  to 
achieve  the  highest  possible  value  of  R  and  therefore  an  inverse  relationship  is  expected, 
but  the  situation  is  further  complicated  by  the  fact  that  users  assign  a  value  judgment  to 
B(U)  based  on  the  utility  of  data  provided.  To  paraphrase  a  popular  aphorism,  ’the  pain 
must  be  worth  the  gain',  and  accurate,  reliable,  useful  data  is  the  expected  output  of  a 
BITE  system.  If  such  a  system  itself  requires  a  significant  amount  of  maintenance,  then 
users  will  tend  to  regard  its  outputs  with  justifiable  skepticism.  The  ideal  case  assumes 
that  R  and  B(U)  are  maximized;  a  perfect  system  would  require  no  maintenance  and 
provide  accurate  indications  at  all  times.  Although  unachievable  in  reality,  this  standard 
apparently  does  provide  a  meaningful  baseline  to  assess  real  BITE  system  performance. 

The  full  data  set,  including  graphics,  is  presented  in  Appendices  1  through  3. 
Unsurprisingly,  given  the  fact  that  in  an  ideal  system  R  and  B(U)  are  equal,  the  system 
with  the  greatest  level  of  user  acceptance  (the  A- 10  TEMS)  exhibits  the  least  significant 
linear  slope  when  these  two  parameters  are  graphically  compared.  What  is  unexpected 
from  a  purely  analytical  viewpoint  is  that  the  least  complex  system  in  terms  of  dedicated 
LRUs  and  diagnostic  resolution  (the  F-15  ASP)  suffers  from  the  lowest  proportional  user 
acceptance  (greatest  slope)  when  its  perceived  reliability  and  maintenance  burden  are 
similarly  compared.  In  other  words,  the  relationship  between  R  and  B(U)  when  viewed 
holistically  across  the  sampled  systems  is  non-linear  if  intrinsic  physical  complexity  and 
M  are  considered  independent  variables.  Clearly,  perception  is  an  integral  component  of 
user  acceptance. 

C.  INTERPRETATION  AND  SIGNIFICANCE 

The  x-axis  in  these  plots  is  dimensionless  and  merely  sequential,  labeling  the 
parameters  in  the  order  in  which  they  are  portrayed  on  the  graph.  Despite  (or  perhaps 
because  of)  this  simplicity,  the  relationship  described  by  Eq.  1  remains  constant,  even 
when  I  is  included  despite  its  apparent  independence  from  the  other  parameters.  This 
strongly  suggests  that  K  is  merely  an  artifact  of  the  initial  selection  of  a  value  of  0.01  for 


18 


M,  which  does  not  lend  any  intrinsic  significance  to  the  constant  or  the  equation. 
However,  this  fact  also  suggests  that  the  relationship  as  expressed  is  significant,  and 
worthy  of  further  investigation. 

Referring  to  Figure  3,  it  is  clear  that  higher  values  of  M  are  correlated  with  lower 
values  of  R,  which  in  turn  is  associated  with  lower  values  of  B(U).  This  ’chain  of 
casualty'  is  difficult  to  express  with  true  mathematical  rigor  because  the  exercise  mixes 
empirical  and  subjective  data.  Nevertheless,  some  tentative  heuristics  can  be  derived  via 
examination  of  the  results  obtained  by  this  study. 

BITE  system  success  depends  on  perceived  system  reliability  by  its  end  users.  The 
C-5  MADAR  system  provides  a  remarkable  array  of  in  situ  troubleshooting  capability  for 
its  users,  yet  is  not  well-received  by  them  due  to  the  fact  that  a  large  fraction  of  their  time 
is  spent  troubleshooting  and  repairing  the  system  itself,  which  tends  to  introduce 
skepticism  when  interpreting  MADAR  alarms.  From  a  design  perspective,  it  is  obvious 
that  the  most  diverse  capabilities  are  completely  secondary  to  system  maintenance 
reliability. 

BITE  system  success  depends  on  data  resolution.  At  the  opposite  end  of  the 
spectrum,  the  F-15  ASP  is  not  well-received  by  its  intended  users  because  the 
information  it  provides  is  non-specific.  Simple  "go/no-go"  indications  for  monitored 
multi-LRU  systems  can  be  easily  dismissed  as  the  result  of  transient  conditions  such  as 
platform  power  fluctuations,  human  error,  etc.,  and  if  such  indications  are  not 
accompanied  by  additional  evidence  of  malfunction  such  as  operator-reported 
discrepancies  then  they  are  usually  ignored  with  long-term  damage  to  BITE  credibility. 
Designers  must  consider  the  practical  utility  of  provided  data  within  the  context  of 
monitored  system's  failure  mode  effects/causes  analyses  (FMECAs)  (Levitt,  2003). 
Specificity  is  critical  for  utility. 

The  number  of  BITE  data  interfaces  is  irrelevant  to  user  acceptance.  This 
heuristic  is  predictive,  but  based  on  acquired  data.  User  trust  is  founded  on  perceived 
reliability,  which  is  a  function  of  both  BITE-provided  data  quality  and  the  BITE  system’s 
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maintenance  burden.  The  ideal  system  model  provides  a  large  number  of  interfaces,  but 
the  model's  output  does  not  change  for  all  values  of  I.  This  characteristic  may  provide 
valuable  guidance  to  designers  as  discussed  in  the  following  sections. 

D.  RESEARCH  QUESTION  ANSWERS 

Although  this  study  is  of  necessity  quite  limited  in  scope,  preliminary  answers  to 
the  stated  research  questions  can  be  derived  from  this  data: 

At  what  point,  and  based  on  which  factors,  does  maintenance  of  built-in-test  and 
monitoring  (BITE)  subsystems  obviate  or  negatively  impact  weapons  system  life-cycle 
cost  and  operational  availability?  Results  strongly  indicated  that  M  is  the  primary  factor 
influencing  weapons  system  life-cycle  costs  associated  with  BITE.  Using  a  standardized 
labor  cost  of  $50.00  per  hour  yields  the  following  average  annual  BITE  system 
maintenance  expenditures  for  each  weapon  system  examined: 

A-10  TEMS:  $263,000  (200  aircraft,  $1315  per  airframe,  M=6.35%) 

C-5  MADAR:  $1,247,293  (94  aircraft,  $13,269  per  airframe,  M=19.07%) 

F-15  ASP:  $421,204  (750  aircraft,  $561  per  airframe,  M=4.26%) 

It  must  be  noted  that  these  are  direct  labor  costs  only  and  do  not  reflect  LRU 
repair  costs  and  the  corresponding  support  infrastructure  needed  to  perform  this  work. 
Total  BITE  system  support  costs  may  in  fact  be  an  order  of  magnitude  (or  more)  higher 
for  complex  systems  such  as  MADAR.  Even  more  importantly,  no  data  exists  to  quantify 
direct  costs  incurred  by  erroneous  LRU  replacement  (and  attendant  unnecessary 
organizational  and  Depot  maintenance)  from  false  BITE  results  or,  conversely,  from 
unnecessary  or  erroneous  troubleshooting  and  component  replacement  performed 
because  technicians  did  not  utilize  or  trust  BITE. 

Intangible  costs  that  are  difficult  to  quantify  may  in  fact  have  a  greater  negative 
impact.  High  values  of  M  were  also  associated  with  low  values  of  R  and  B(U),  implying 
that  technicians  were  not  convinced  of  the  system's  accuracy  as  a  troubleshooting  tool. 
Based  on  the  author's  experience,  this  paradigm  frequently  results  in  extensive  additional 
troubleshooting  of  monitored  aircraft  system  malfunctions  as  well  as  erroneous 
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component  replacement.  Therefore,  a  BITE  system  M  value  can  serve  as  a  reasonably 
reliable  indicator  of  user  acceptance  and  therefore  system  effectiveness,  and  M  is  in  turn 
directly  related  to  total  system  operational  availability.  Low  values  of  M  therefore 
increase  A0  and  decrease  total  life-cycle  costs. 

Is  there  an  empirical  relationship  between  the  overall  systemic  complexity  of 
BITE  subsystems,  the  analogous  complexity  of  monitored  subsystems,  and  total  life-cycle 
cost  for  a  given  in  tegrated  weapons  system  so  equipped?  Such  a  relationship  does  appear 
to  exist,  although  ’empirical’  may  be  too  strong  a  tenn  within  the  findings  of  this  study; 
stochastic  is  a  more  accurate  description.  The  most  surprising  result  was  that  the  number 
of  interfaces  (I)  appears  to  have  no  effect  on  M,  which  is  counterintuitive.  More  detailed 
research  efforts  are  needed  to  more  fully  understand  and  quantify  the  large  number  of 
secondary  variables  involved.  It  is  tempting  to  speculate  that  more  effective  BITE  self¬ 
diagnostics  would  improve  M,  but  this  conjecture  if  pursued  to  its  logical  extreme  might 
easily  devolve  into  an  endless  amount  of  increasingly  detailed  internal  self-monitoring, 
which  at  some  point  would  act  to  increase  M  as  even  more  transient  conditions  generate 
operator  alarms. 

Can  a  mathematical  model  representing  interfaces  and  monitoring  capabilities  be 
constructed  to  optimize  BITE  infrastructure  during  the  design  phase  to  provide  maximum 
operational  availability  while  minimizing  life-cycle  costs?  Optimization  in  the  sense 
originally  envisioned  at  the  beginning  of  this  work  does  not  seem  feasible.  However,  the 
analysis  did  define  M  to  reveal  it  as  the  most  critical  design  parameter  for  system  success 
due  to  its  direct  correlation  with  measures  of  user  trust  and  independence  from  the 
number  of  data  interfaces. 
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III.  RESULTS  AND  RECOMMENDATIONS 


A.  RECOMMENDATIONS  FOR  FUTURE  DESIGNS 

The  prime  heuristic  derived  is  that  dedicated  BITE  systems  must  minimize  user 
maintenance  burden  in  order  to  gain  acceptance  and  fulfill  the  system's  design  objectives. 
Despite  the  demonstrated  independence  of  the  number  of  interfaces  to  M,  it  is  obvious 
that  most  malfunctions  will  occur  at  such  interfaces,  particularly  in  electrical  connectors 
and  other  areas  of  discontinuity  in  signal  paths  that  are  exposed  to  external  environmental 
effects.  Therefore,  implementing  multi-path  and  multi-sensor  redundancy  in  a  manner 
similar  to  that  employed  for  safety-critical  systems  such  as  automatic  pilots  (Branch, 
1998)  would  be  a  most  effective  strategy  to  assure  that  a  given  BITE  architecture  wins 
user  acceptance.  For  example,  the  C-5  MADAR  engine  vibration  monitoring  suite  could 
be  augmented  by  adding  another  full  set  of  sensors  to  each  engine,  thus  providing  an 
inherent  "reality  check"  to  sensor  readings  and  preventing  or  minimizing 
unnecessary/erroneous  maintenance,  which  in  this  instance  can  include  full  replacement 
of  suspect  engines. 

An  important  secondary  finding  was  that  data  resolution  and  return  was  positively 
correlated  with  user  acceptance,  though  modulated  by  perceived  system  reliability.  As 
previously  discussed,  the  F- 15  ASP  'go/no-go'  indication  method  apparently  did  not  yield 
sufficiently  decisive  data  to  win  confidence  in  the  veracity  of  the  results.  Conversely,  the 
C-5  MADAR  provides  not  only  threshold  exceedance  alarms  but  also  real-time  data 
usually  presented  in  the  form  of  0  to  5  volt  DC  conditioned  and  scaled  signal  levels. 
Despite  this  unusually  versatile  troubleshooting  capability,  the  system  suffers  from  poor 
user  acceptance;  in  the  author's  opinion,  the  sheer  number  of  MADAR  alarms  commonly 
generated  leads  many  technicians  to  disregard  them  as  transient  phenomena  in  much  the 
same  way  that  the  F-15  ASP  panel  is  seen  as  irrelevant  in  some  respects,  and  leads  to  a 
dismissive  attitude  towards  the  system's  capabilities. 

The  most  successful  system  examined,  the  A- 10  TEMS,  provides  not  only 
recorded  in-flight  data  for  later  analysis  but  also  (via  use  of  secondary  ground  support 
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equipment  commonly  known  as  the  digital  display  unit,  or  DDU)  the  ability  to  examine 
real-time  data  presented  in  engineering  units.  This  design  philosophy  seems  to  increase 
user  acceptance  in  two  fundamental  ways.  Since  the  DDU  is  portable,  it  accuracy  can  be 
readily  checked  by  connecting  the  device  to  another  aircraft,  or  attaching  a  different  unit 
to  a  suspect  aircraft.  Secondly,  the  DDU's  displays  closely  correspond  with  those  of  the 
systems  monitored  by  TEMS,  the  aircraft  engines;  interpolation  of  TEMS  data  is  greatly 
minimized  thereby. 

A  corollary  to  the  prime  heuristic  that  dedicated  BITE  systems  must  minimize 
user  maintenance  burden  in  order  to  gain  acceptance  and  fulfill  the  system's  design 
objectives  is  that  increased  development  and  procurement  costs  needed  to  provide  the 
requisite  sensor  redundancy  will  yield  reduced  sustainment  costs  over  the  lifetime  of  the 
weapons  system.  Due  to  highly  variable  budget  environments,  most  Department  of 
Defense  systems  across  all  mission  areas  usually  remain  in  operational  use  well  beyond 
their  original  design  lifetimes.  Additionally,  equipment  such  as  space  mission  control 
systems  are  often  designed  for  "backwards  compatibility"  with  legacy  satellite 
constellations.  A  recent  example  is  the  Advanced  Extremely  High  Frequency  (AEHF) 
Satellite  Mission  Control  System  (ASMCS),  which  will  replace  the  Milstar  Satellite 
Mission  Control  System  (SMCS)  (Branham,  2006).  Accurate  BITE  capability  that  is 
trusted  implicitly  by  its  users  is  urgently  needed  both  due  to  the  nature  of  the  equipment's 
function  and  the  need  to  restore  it  quickly  as  well  as  the  fact  that  such  a  system  must 
interface  with  two  or  more  generations  of  technology  in  terms  of  spacecraft  controlled.  In 
the  AEHF  example,  the  need  to  distinguish  latent  interface  incompatibilities,  software 
problems,  and  operator  errors  from  genuine  equipment  malfunctions  is  overwhelmingly 
critical  and  time-sensitive  during  both  peacetime  and  wartime  military  operations  since 
potential  loss  of  a  strategically  essential  communications  satellite  would  have  a 
devastating  impact  on  all  DoD  functions. 

B.  IMPLICATIONS  FOR  ADVANCED  BITE  ARCHITECTURES 

Condition-based  maintenance  (CBM)  is  an  emerging  concept  for  most  DoD 
weapons  systems.  The  central  concept  is  to  continuously  monitor  subsystem  performance 
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and  execute  preventative  maintenance  actions  prior  to  the  occurrence  of  non-mission- 
capable  failures  (US  Department  of  Defense,  2006).  Aircraft  engine  trend  monitoring  in 
the  commercial  sector  is  an  early  implementation  of  CBM,  and  USAF  is  deploying  a 
more  comprehensive  architecture  known  as  CFRS  (computerized  fault  reporting  system) 
which  will  monitor  the  F-15  and  C-130  fleets.  Clearly,  the  goals  of  CBM  cannot  be 
realized  if  platform  BITE  lacks  reliability  and  incurs  a  large  maintenance  burden,  or,  in 
other  words,  exhibits  a  high  value  of  M.  Not  only  would  consequent  low  levels  of  user 
acceptance  eventually  drive  such  systems  into  limited  use,  but  overall  sustainment  costs 
would  increase  disproportionately,  especially  if  ancillary  technical  data  is  not  procured 
due  to  mandates  to  use  perceived— at  least  at  higher  echelons— infallible  BITE  systems, 
and  consequently  lower  levels  of  training  are  provided  to  end  users  in  a  misguided  belief 
that  a  given  weapon  system  "can  troubleshoot  itself'. 

Arguably,  this  situation  is  the  case  with  the  C-5  MADAR.  Despite 
troubleshooting  trees  that  can  stretch  into  literally  hundreds  of  steps,  the  system  if  used 
mechanistically  as  intended  in  accordance  with  published  technical  data  absent  of 
inherent  experience  and  judgment  rarely  produces  decisive  conclusions.  Indeed,  the  sheer 
magnitude  of  effort  required  to  perform  these  procedures  verbatim  frequently  discourages 
technicians  from  performing  them  in  their  entirety,  and  therefore  again  defeats  the 
functional  utility  of  the  system. 

Sensor  redundancy  in  proportion  to  monitored  function  criticality  combined  with 
parsimonious  selection  of  other  subsidiary  monitored  subsystems  is  therefore  the  only 
reasonable  design  strategy  apparent  to  avoid  these  highly  undesirable  circumstances  and 
consequences. 

These  design  observations  may  yield  significant  improvement  in  future  BITE 

architectures.  However,  one  of  the  most  intractable  problems  revealed  by  the  study  was 

the  fact  that  users  and  maintainers  of  BITE  are  often  the  very  same  individuals,  which 

conceivably  can  incur  a  lack  of  oversight  or  objective  evaluation  in  terms  of  system 

utilization  and  performance.  It  would  be  absurd  to  establish  a  completely  distinct  Air 

Force  Specialty  Code  (AFSC)  or  Military  Occupational  Specialty  (MOS)  for  other 

services  dedicated  to  BITE  system  maintenance,  but  some  level  of  separation  must  be 
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achieved  between  the  user  and  operator  roles  in  order  to  decouple  them  and  prevent 
undesirable  negative  feedback  loops  to  become  established.  One  means  of  doing  so 
would  be  to  dedicate  selected  maintenance  personnel  to  BITE  maintenance  only  for 
limited  periods.  This  method  would  both  concentrate  system  knowledge  for  the  detailed 
personnel  and  provide  objective,  rather  than  subjective,  analysis  of  malfunctions, 
hopefully  resulting  in  improved  user  confidence  and  subsequently  better  BITE  system 
utilization. 

The  only  practical  alternative  to  BITE  is  specialized  test  equipment,  from  the 
perspective  of  system  maintainers.  Such  devices  themselves  require  extensive  logistics 
support  throughout  their  life-cycle,  and  therefore  military  acquisition  efforts  generally 
tend  to  minimize  development  of  this  equipment.  Furthermore,  test  equipment  can  be 
cumbersome  to  use,  and  this  characteristic  therefore  may  tend  to  discourage  its 
employment. 

From  the  perspective  of  system  operators  such  as  aircrew  members  and  satellite 
controllers,  the  unique  ability  of  BITE  to  provide  immediate  verification  of  system  status 
is  invaluable,  and  cannot  be  replaced  by  any  other  schema.  It  is  essential  for  users  to  have 
some  indication  of  system  status  before  committing  to  operational  decisions,  and  external 
test  equipment  cannot  satisfy  this  requirement. 

C.  CONCLUSION 

Although  passive  stand-alone  BITE  may  eventually  prove  to  be  a  transient  phase 
in  weapon  system  design,  BITE  capability  as  a  paradigm  is  here  to  stay,  and  the  lessons 
learned  in  this  study  are  equally  applicable  to  such  alternative  architectures.  There  is  a 
relatively  fine  line  between  truly  adding  value  and  merely  adding  unnecessary  labor  and 
sustainment  expense  through  employment  of  such  systems,  defined  almost  exclusively  by 
the  derived  indicator  M. 

Methods  for  obtaining  lower  values  of  M  must  include  both  physical  design 
considerations  and  development/adoption  of  user  policies  intended  to  minimize  the  field- 
level  maintenance  burden.  As  previously  discussed,  redundant  sensors  for  critical 
platform  data  acquisition  would  allow  cross-checking  of  indicated  malfunctions  and 
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thereby  improve  user  confidence  in  BITE  results.  However,  adding  interfaces  and 
equipment  must  also  increase  the  potential  for  local  system  malfunctions  and  increase  the 
system’s  maintenance  burden. 

From  a  policy  perspective,  if  critical  sensors  are  triple  or  quadruple-redundant 
then  repairs  of  single  or  double  sensor  failures  could  be  deferred  until  the  next  periodic 
Depot  maintenance  period.  Removing  this  additional  M  contribution  from  the  field  user's 
area  of  responsibility  would  facilitate  a  refocus  on  the  subject  BITE  system’s  core 
functionality  and  therefore  improve  overall  platfonn  maintenance. 

For  applications  such  as  unmanned  spacecraft  ground  control,  redundant  fault 
detection  architectures  (software  or  hardware  based)  are  even  more  critical.  Restoring 
functionality  as  rapidly  as  possible  is  of  paramount  importance  and  is  operationally 
equivalent  to  an  aircrew  reacting  to  an  in-flight  emergency.  Despite  the  fact  that  there  is 
no  potential  for  direct  loss  of  life  in  these  situations,  loss  of  a  space  vehicle  usually 
cannot  easily  or  rapidly  be  redressed,  and  such  losses  invariably  incur  a  large  negative 
impact  on  the  system's  user  community. 

Additionally,  simplifying  troubleshooting  procedures  to  the  greatest  degree 
possible  would  again  make  a  significant  contribution  to  BITE  employment.  Onerous, 
lengthy  procedures  are  obviously  non-quantifiable  but  important  negative  influences  on 
system  acceptance  that  may  act  to  increase  M  based  on  the  author's  experience,  since 
'collateral'  findings  of  BITE  anomalies  frequently  occur  during  such  efforts. 

These  suggestions  are  only  some  of  the  possible  applications  of  the  findings  of 
this  research.  Further  investigation  is  likely  to  reveal  additional  methods  to  refine  BITE 
system  performance  from  a  holistic  perspective.  The  main  conclusion  of  this  study  is  that 
obtaining  values  of  M  as  close  as  possible  (or  ideally  lower)  than  0.01  will  provide  truly 
useful  BITE  performance  and  greatly  reduce  sustainment  costs  over  the  operational 
lifetime  of  DoD  weapon  systems. 
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APPENDIX  A.  A-10  TEMS  DATA 


Table  3.  A-10  REMIS  data(April  1987-April  2007). 


System 

Maintenance  Hours 

01 

0.0 

02 

0.0 

03 

0.0 

04 

0.0 

05 

0.0 

06 

0.0 

07 

0.0 

09 

0.0 

11 

721433.6 

12 

250983.1 

13 

689904.7 

14 

757789.8 

23 

647693.9 

24 

84716.0 

41 

176759.6 

42 

259270.7 

44 

121131.8 

45 

125130.5 

46 

576997.6 

47 

67975.1 

49 

30740.3 

51 

301701.3 

52 

58629.1 

55 

105201.3 

62 

98719.7 

63 

98126.5 

64 

28226.4 

65 

69180.3 

69 

24081.2 

71 

108180.9 

72 

10109.8 

74 

496014.5 

75 

764069.6 

76 

346090.7 

91 

16935.6 

96 

938.7 

97 

62197.5 

99 

121.9 

AA 

47.6 

AP 

13.0 

XX 

32.0 
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Appendix  A.  A- 10  data  (continued). 
Table  4.  TEMS  survey  results. 


N=9 

A 

B 

c 

D 

E 

Question  1 

4 

2 

1 

1 

1 

Question  2 

4 

1 

4 

Question  3 

1 

2 

5 

1 

Question  4 

1 

3 

4 

1 

Question  5 

6 

2 

1 

Normalization  (A 
E=l) 

=5,  B=4,  C=3, 

D=2, 

Averages 

Comments 

A 

B  C 

D 

E 

>20%  time  resolving  TEMS 

20 

8  3 

2 

1 

3.78 

discrepancies 

Sometimes  use  TEMS  to  troubleshoot 

0 

16  3 

8 

0 

3.00 

other  systems 

5 

8  15 

2 

0 

3.33 

Medium  trust  in  system  accuracy 

5 

12  12 

0 

1 

3.33 

Medium  repair  difficulty 

Most  respondents  have  no  experience 

30 

8  3 

0 

0 

4.56 

on  other  acft  types 
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Appendix  A.  A- 10  data  (continued). 


Figure  7.  A-lOMvs.  I. 


A-10  M  vs.  R 


Figure  8.  A-10  M  vs.  R. 
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A-10  M  vs.  B(U) 


Figure  9.  A-10  M  vs.  B(U). 


Figure  10.  A-10  R  vs.  B(U). 
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APPENDIX  B.  C-5  MADAR  DATA 


Table  5.  C-5  REMIS  data  (April  1987-April  2007). 


C-5A  C-5B  C-5C 


System 

Maint.  Hrs. 

System 

Maint.  Hrs. 

System 

Maint.  Hrs. 

01 

0.0 

01 

0.0 

01 

0.0 

02 

0.0 

02 

0.0 

02 

0.0 

03 

0.0 

03 

0.0 

03 

0.0 

04 

0.0 

04 

0.0 

04 

0.0 

05 

0.0 

05 

0.0 

06 

0.0 

06 

0.0 

06 

0.0 

07 

0.0 

07 

0.0 

07 

0.0 

09 

0.0 

09 

0.0 

09 

0.0 

11 

34914.7 

11 

1581203.5 

11 

721828.6 

12 

6452.4 

12 

518756.8 

12 

226380.1 

13 

30347.3 

13 

1155500.8 

13 

697184.1 

14 

17146.7 

14 

681976.2 

14 

425145.3 

23 

29783.0 

23 

1249294.8 

23 

725908.0 

24 

3213.1 

24 

129121.0 

24 

85663.0 

41 

7602.4 

41 

424502.6 

41 

213185.1 

42 

5305.5 

42 

263791.5 

42 

172091.0 

44 

6240.1 

44 

333337.4 

44 

185115.7 

45 

5585.9 

45 

346273.0 

45 

177092.4 

46 

9005.6 

46 

545152.0 

46 

273657.3 

47 

820.0 

47 

84449.3 

47 

36360.2 

49 

5852.7 

49 

256473.0 

49 

151935.0 

51 

5454.8 

51 

265483.3 

51 

164570.0 

52 

4648.0 

52 

276178.7 

52 

161455.0 

55 

5885.0 

55 

332546.6 

55 

160485.8 

57 

21.3 

57 

121.8 

57 

1794.5 

59 

1001.7 

59 

28623.0 

59 

25107.7 

61 

716.8 

61 

52808.6 

61 

27420.7 

62 

193.6 

62 

26729.5 

62 

18124.0 

63 

63 

27685.5 

63 

14981.8 

64 

64 

122614.7 

64 

72570.2 

65 

65 

37205.0 

65 

20948.4 

66 

927.4 

66 

54177.5 

66 

20528.9 

68 

6.0 

68 

559.8 

68 

512.7 

69 

17.8 

69 

3226.4 

69 

1850.6 

71 

1010.2 

71 

44173.3 

71 

22238.4 

72 

3793.9 

72 

209716.5 

72 

142205.4 

76 

22.3 

76 

607.2 

76 

27082.0 

82 

149.1 

82 

26.2 

82 

3316.8 

91 

652.8 

91 

103918.9 

91 

51981.5 

97 

416.3 

97 

24730.7 

97 

9356.4 

AA 

1.0 

Y5 

433.0 

Y5 

18.8 

ZZ 

2.0 
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Appendix  B.  C-5  data  (continued). 


Table  6.  MADAR  survey  results. 


N=ll 

A 

B 

C 

D 

E 

Question  1 

2 

6 

1 

2 

Question  2 

1 

3 

5 

2 

Question  3 

5 

5 

1 

Question  4 

1 

3 

7 

Question  5 

2 

2 

1 

4 

2 

Normalization 
C=3,  D=2,  E=l) 

(A=5, 

B=4, 

Averages 

Comments 

A  B 

C 

D 

E 

>20%  time  resolving  MADAR 

10  24 

3 

4 

0 

3.73 

discrepancies 

Often  use  MADAR  to  troubleshoot  other 

0  4 

9 

10 

2 

2.27 

systems 

0  0 

15 

10 

1 

2.36 

Medium/low  trust  in  system  accuracy 

5  0 

9 

14 

0 

2.55 

Medium/low  repair  difficulty 

Most  respondents  have  experience  on  at 

10  8 

3 

8 

2 

2.82 

least  2  other  acft  types 
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Appendix  B.  C-5  data  (continued). 


Figure  11.  C-5  M  vs.  I 


Figure  12.  C-5  M  vs.  R. 
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Figure  13.  C-5  M  vs.  B(U). 


Figure  14.  C-5  R  vs.  B(U). 
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APPENDIX  C.  F-15  ASP  DATA 


Table  7.  F-15  REMIS  data  (April  1987-April  2007). 

F-15A  F-15B 


01 

0.0 

02 

0.0 

03 

0.0 

04 

0.0 

05 

0.0 

06 

0.0 

07 

0.0 

09 

0.0 

11 

484804.8 

12 

127919.6 

13 

314846.8 

14 

233442.8 

1A 

0.0 

23 

697271.8 

24 

258583.1 

28 

7.3 

41 

160550.3 

42 

110542.7 

44 

69525.8 

45 

152660.3 

46 

366771.8 

47 

25327.1 

49 

43579.9 

51 

146351.6 

52 

86688.8 

55 

30628.9 

57 

52963.1 

61 

1891.9 

63 

113995.4 

65 

86388.3 

71 

130026.8 

72 

8.5 

74 

679123.4 

75 

240319.7 

76 

351390.4 

91 

2774.7 

92 

605.4 

97 

39918.6 

99 

1828.5 

01 

0.0 

02 

0.0 

03 

0.0 

04 

0.0 

05 

0.0 

06 

0.0 

07 

0.0 

09 

0.0 

11 

123312.9 

12 

50569.3 

13 

89792.7 

14 

60132.3 

23 

127040.5 

24 

50023.6 

41 

37188.0 

42 

26040.3 

44 

17876.6 

45 

36551.8 

46 

73099.6 

47 

8517.8 

49 

6575.8 

51 

37275.5 

52 

18160.8 

53 

0.2 

55 

5592.3 

57 

13258.9 

61 

400.9 

63 

30845.3 

65 

16743.4 

71 

27455.5 

72 

92.0 

74 

135221.0 

75 

54599.1 

76 

28122.3 

91 

1865.8 

92 

239.1 

97 

16680.3 

99 

3388.0 
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Appendix  C.  F-15  data  (continued). 


F-15C 


01 

0.0 

02 

0.0 

03 

0.0 

04 

0.0 

05 

0.0 

06 

0.0 

07 

0.0 

08 

0.0 

09 

0.0 

11 

1702360.9 

12 

431346.1 

13 

1173499.4 

14 

825112.9 

23 

1313987.9 

24 

831837.4 

28 

186.1 

41 

524290.7 

42 

495251.2 

44 

321795.9 

45 

466743.7 

46 

1156392.7 

47 

140682.4 

49 

70847.8 

51 

408060.6 

52 

281959.1 

53 

87.8 

55 

111710.6 

57 

128623.4 

61 

181.8 

63 

413791.1 

65 

272157.7 

71 

449134.2 

72 

134.2 

74 

2573993.1 

75 

1060629.4 

76 

1954716.7 

77 

24.2 

82 

185.8 

91 

16204.9 

92 

504.6 

97 

77904.9 

99 

625.5 

AA 

22.5 

ZZ 

107.2 

F-15D 


01 

0.0 

02 

0.0 

03 

0.0 

04 

0.0 

05 

0.0 

06 

0.0 

07 

0.0 

09 

0.0 

11 

352467.8 

12 

134693.5 

13 

228223.5 

14 

157345.1 

21 

0.0 

23 

240091.7 

24 

134408.3 

28 

7.0 

41 

81740.1 

42 

84566.7 

44 

57066.6 

45 

77931.9 

46 

183126.5 

47 

28119.4 

49 

9854.1 

51 

80349.1 

52 

48232.3 

53 

436.9 

55 

20549.9 

57 

23518.1 

61 

104.8 

63 

95276.3 

65 

41720.4 

71 

77712.7 

72 

15.0 

74 

413003.9 

75 

168492.6 

76 

157290.0 

77 

38.0 

82 

12.0 

91 

4993.2 

92 

445.3 

97 

23750.1 

99 

911.1 

AA 

0.0 

ZZ 

61.2 

38 


Appendix  C.  F-15  data  (continued). 


Table  8. 

ASP 

survey 

results. 

N=19 

A 

B 

C 

D 

Question  1 

11 

4 

2 

1 

Question  2 

2 

2 

10 

4 

Question  3 

1 

6 

8 

4 

Question  4 

1 

8 

3 

Question  5  7 

Normalization  (A= 
03,  D=2,  E=l) 

9 

5,  B=4, 

1 

Averages 

55  16 

6 

2 

1 

4.21 

10  8 

30 

8 

1 

3.00 

5  24 

24 

8 

0 

3.21 

0  4 

24 

6 

7 

2.16 

35  36 

3 

0 

2 

4.00 

E 

1 

1 

7 

2 

Result 

<20%  time  resolving  ASP  discrepancies 

Sometimes  use  ASP  to  troubleshoot  other 
systems 

Low/medium  trust  in  system  accuracy 

Low  repair  difficulty 

Most  respondents  have  experience  on  at 
least  one  other  acft  type 
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Appendix  C.  F-15  data  (continued). 


Figure  15.  F-15  M  vs.  I. 


F-15  M  vs.  R 


Figure  16.  F-15  M  vs.  R. 
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F-15Mvs.  B(U) 


Figure  17.  F-15  M  vs.  B(U). 


Figure  18.  F-15  R  vs.  B(U). 
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APPENDIX  D.  SYSTEM  SURVEY 


[SUBJECT  AIRCRAFT/SYSTEM]  QUESTIONNAIRE 
Ver  2-  24  Apr  07 


1.  If  you  are  an  avionics  technician,  how  much  of  your  time  doing  avionics 
work  is/was  spent  on  [subject  system]  discrepancies? 

a.  0%-20%  c.  40%-60%  e.  80%-100% 

b.  20%-40%  d.  60%-80% 


2.  How  often  do  you  use  [subject  system]  to  troubleshoot  other  aircraft 

systems? 

a.  Never  c.  Sometimes  e.  Very  often 

b.  Rarely  d.  Often 


3.  How  much  do  you  trust  [subject  system]  readings  when  troubleshooting  other 

systems? 

a.  Not  at  all  c.  Somewhat  e.  Completely 

b.  A  little  d.  Quite  a  bit 


4.  Compared  to  other  [subject  aircraft]  avionics  systems,  how  difficult  is  [subject 
system]  to  troubleshoot  &  repair? 

a.  Much  more  c.  Medium/no  difference  e.  Easy 

b.  A  little  more  d.  Not  as  bad  as  some  others 


5.  How  many  other  types  of  aircraft  have  you  worked? 

a.  Zero  c.  Two  e.  Four  or  more 

b.  One  d.  Three 


Please  write  your  rank  and  AFSC,  including  skill  level  (3,  5,  7)-  names  aren’t  needed-and 
return  this  to  Nick  Previsich.  Thanks! 
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